Statistical Reliability
   HOME

TheInfoList



OR:

In
statistics Statistics (from German language, German: ''wikt:Statistik#German, Statistik'', "description of a State (polity), state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of ...
, a consistent estimator or asymptotically consistent estimator is an
estimator In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguished. For example, the ...
—a rule for computing estimates of a parameter ''θ''0—having the property that as the number of data points used increases indefinitely, the resulting sequence of estimates
converges in probability In probability theory, there exist several different notions of convergence of random variables. The convergence of sequences of random variables to some limit random variable is an important concept in probability theory, and its applications to ...
to ''θ''0. This means that the distributions of the estimates become more and more concentrated near the true value of the parameter being estimated, so that the probability of the estimator being arbitrarily close to ''θ''0 converges to one. In practice one constructs an estimator as a function of an available sample of
size Size in general is the Magnitude (mathematics), magnitude or dimensions of a thing. More specifically, ''geometrical size'' (or ''spatial size'') can refer to linear dimensions (length, width, height, diameter, perimeter), area, or volume ...
''n'', and then imagines being able to keep collecting data and expanding the sample ''ad infinitum''. In this way one would obtain a sequence of estimates indexed by ''n'', and consistency is a property of what occurs as the sample size “grows to infinity”. If the sequence of estimates can be mathematically shown to converge in probability to the true value ''θ''0, it is called a consistent estimator; otherwise the estimator is said to be inconsistent. Consistency as defined here is sometimes referred to as ''weak consistency''. When we replace convergence in probability with
almost sure convergence In probability theory, there exist several different notions of convergence of random variables. The convergence of sequences of random variables to some limit random variable is an important concept in probability theory, and its applications to ...
, then the estimator is said to be ''strongly consistent''. Consistency is related to
bias Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group, ...
; see bias versus consistency.


Definition

Formally speaking, an
estimator In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguished. For example, the ...
''Tn'' of parameter ''θ'' is said to be consistent, if it
converges in probability In probability theory, there exist several different notions of convergence of random variables. The convergence of sequences of random variables to some limit random variable is an important concept in probability theory, and its applications to ...
to the true value of the parameter: : \underset\;T_n = \theta. i.e. if, for all ''ε'' > 0 : \lim_\Pr\big(, T_n-\theta, > \varepsilon\big) = 0. A more rigorous definition takes into account the fact that ''θ'' is actually unknown, and thus the convergence in probability must take place for every possible value of this parameter. Suppose is a family of distributions (the
parametric model In statistics, a parametric model or parametric family or finite-dimensional model is a particular class of statistical models. Specifically, a parametric model is a family of probability distributions that has a finite number of parameters. Def ...
), and is an infinite
sample Sample or samples may refer to: Base meaning * Sample (statistics), a subset of a population – complete data set * Sample (signal), a digital discrete sample of a continuous analog signal * Sample (material), a specimen or small quantity of s ...
from the distribution ''pθ''. Let be a sequence of estimators for some parameter ''g''(''θ''). Usually ''Tn'' will be based on the first ''n'' observations of a sample. Then this sequence is said to be (weakly) consistent if : \underset\;T_n(X^) = g(\theta),\ \ \text\ \theta\in\Theta. This definition uses ''g''(''θ'') instead of simply ''θ'', because often one is interested in estimating a certain function or a sub-vector of the underlying parameter. In the next example we estimate the location parameter of the model, but not the scale:


Examples


Sample mean of a normal random variable

Suppose one has a sequence of
statistically independent Independence is a fundamental notion in probability theory, as in statistics and the theory of stochastic processes. Two events are independent, statistically independent, or stochastically independent if, informally speaking, the occurrence of o ...
observations from a normal ''N''(''μ'', ''σ''2) distribution. To estimate ''μ'' based on the first ''n'' observations, one can use the
sample mean The sample mean (or "empirical mean") and the sample covariance are statistics computed from a Sample (statistics), sample of data on one or more random variables. The sample mean is the average value (or mean, mean value) of a sample (statistic ...
: ''Tn'' = (''X''1 + ... + ''Xn'')/''n''. This defines a sequence of estimators, indexed by the sample size ''n''. From the properties of the normal distribution, we know the
sampling distribution In statistics, a sampling distribution or finite-sample distribution is the probability distribution of a given random-sample-based statistic. If an arbitrarily large number of samples, each involving multiple observations (data points), were s ...
of this statistic: ''T''''n'' is itself normally distributed, with mean ''μ'' and variance ''σ''2/''n''. Equivalently, \scriptstyle (T_n-\mu)/(\sigma/\sqrt) has a standard normal distribution: : \Pr\!\left T_n-\mu, \geq\varepsilon\,\right= \Pr\!\left \frac \geq \sqrt\varepsilon/\sigma \right= 2\left(1-\Phi\left(\frac\right)\right) \to 0 as ''n'' tends to infinity, for any fixed . Therefore, the sequence ''Tn'' of sample means is consistent for the population mean ''μ'' (recalling that \Phi is the
cumulative distribution In statistics, the frequency (or absolute frequency) of an event i is the number n_i of times the observation has occurred/recorded in an experiment or study. These frequencies are often depicted graphically or in tabular form. Types The cumul ...
of the normal distribution).


Establishing consistency

The notion of asymptotic consistency is very close, almost synonymous to the notion of convergence in probability. As such, any theorem, lemma, or property which establishes convergence in probability may be used to prove the consistency. Many such tools exist: * In order to demonstrate consistency directly from the definition one can use the inequality :: \Pr\!\big (T_n-\theta)\geq\varepsilon\big\leq \frac, the most common choice for function ''h'' being either the absolute value (in which case it is known as
Markov inequality In probability theory, Markov's inequality gives an upper bound for the probability that a non-negative function (mathematics), function of a random variable is greater than or equal to some positive Constant (mathematics), constant. It is named a ...
), or the quadratic function (respectively
Chebyshev's inequality In probability theory, Chebyshev's inequality (also called the Bienaymé–Chebyshev inequality) guarantees that, for a wide class of probability distributions, no more than a certain fraction of values can be more than a certain distance from th ...
). * Another useful result is the
continuous mapping theorem In probability theory, the continuous mapping theorem states that continuous functions preserve limits even if their arguments are sequences of random variables. A continuous function, in Heine’s definition, is such a function that maps converg ...
: if ''Tn'' is consistent for ''θ'' and ''g''(·) is a real-valued function continuous at point ''θ'', then ''g''(''Tn'') will be consistent for ''g''(''θ''): :: T_n\ \xrightarrow\ \theta\ \quad\Rightarrow\quad g(T_n)\ \xrightarrow\ g(\theta) * Slutsky’s theorem can be used to combine several different estimators, or an estimator with a non-random convergent sequence. If ''Tn'' →''d''''α'', and ''Sn'' →''p''''β'', then :: \begin & T_n + S_n \ \xrightarrow\ \alpha+\beta, \\ & T_n S_n \ \xrightarrow\ \alpha \beta, \\ & T_n / S_n \ \xrightarrow\ \alpha/\beta, \text\beta\neq0 \end * If estimator ''Tn'' is given by an explicit formula, then most likely the formula will employ sums of random variables, and then the
law of large numbers In probability theory, the law of large numbers (LLN) is a theorem that describes the result of performing the same experiment a large number of times. According to the law, the average of the results obtained from a large number of trials shou ...
can be used: for a sequence of random variables and under suitable conditions, :: \frac\sum_^n g(X_i) \ \xrightarrow\ \operatorname ,g(X)\,/math> * If estimator ''Tn'' is defined implicitly, for example as a value that maximizes certain objective function (see
extremum estimator In statistics and econometrics, extremum estimators are a wide class of estimators for parametric models that are calculated through maximization (or minimization) of a certain objective function, which depends on the data. The general theory of ...
), then a more complicated argument involving
stochastic equicontinuity In estimation theory in statistics, stochastic equicontinuity is a property of estimators (estimation procedures) that is useful in dealing with their asymptotic behaviour as the amount of data increases. It is a version of equicontinuity used in t ...
has to be used.


Bias versus consistency


Unbiased but not consistent

An estimator can be
unbiased Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group, ...
but not consistent. For example, for an
iid In probability theory and statistics, a collection of random variables is independent and identically distributed if each random variable has the same probability distribution as the others and all are mutually independent. This property is us ...
sample one can use ''T''(''X'') = ''x'' as the estimator of the mean E 'X'' Note that here the sampling distribution of ''T'' is the same as the underlying distribution (for any ''n,'' as it ignores all points but the last), so E 'T''(''X'')= E 'X''and it is unbiased, but it does not converge to any value. However, if a sequence of estimators is unbiased ''and'' converges to a value, then it is consistent, as it must converge to the correct value.


Biased but consistent

Alternatively, an estimator can be biased but consistent. For example, if the mean is estimated by \sum x_i + it is biased, but as n \rightarrow \infty, it approaches the correct value, and so it is consistent. Important examples include the
sample variance In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its population mean or sample mean. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbe ...
and
sample standard deviation In statistics, the standard deviation is a measure of the amount of variation or dispersion of a set of values. A low standard deviation indicates that the values tend to be close to the mean (also called the expected value) of the set, while ...
. Without
Bessel's correction In statistics, Bessel's correction is the use of ''n'' − 1 instead of ''n'' in the formula for the sample variance and sample standard deviation, where ''n'' is the number of observations in a sample. This method corrects the bias in t ...
(that is, when using the sample size n instead of the degrees of freedom n-1), these are both negatively biased but consistent estimators. With the correction, the corrected sample variance is unbiased, while the corrected sample standard deviation is still biased, but less so, and both are still consistent: the correction factor converges to 1 as sample size grows. Here is another example. Let T_n be a sequence of estimators for \theta. :\Pr(T_n) = \begin 1 - 1/n, & \mbox\, T_n = \theta \\ 1/n, & \mbox\, T_n = n\delta + \theta \end We can see that T_n \xrightarrow \theta, \operatorname _n= \theta + \delta , and the bias does not converge to zero.


See also

*
Efficient estimator In statistics, efficiency is a measure of quality of an estimator, of an experimental design, or of a hypothesis testing procedure. Essentially, a more efficient estimator, needs fewer input data or observations than a less efficient one to achie ...
*
Fisher consistency In statistics, Fisher consistency, named after Ronald Fisher, is a desirable property of an estimator asserting that if the estimator were calculated using the entire population rather than a sample, the true value of the estimated parameter would ...
— alternative, although rarely used concept of consistency for the estimators *
Regression dilution Regression dilution, also known as regression attenuation, is the Bias (statistics), biasing of the linear regression regression slope, slope towards zero (the underestimation of its absolute value), caused by errors in the independent variable. ...
*
Statistical hypothesis testing A statistical hypothesis test is a method of statistical inference used to decide whether the data at hand sufficiently support a particular hypothesis. Hypothesis testing allows us to make probabilistic statements about population parameters. ...
*
Instrumental variables estimation In statistics, econometrics, epidemiology and related disciplines, the method of instrumental variables (IV) is used to estimate causal relationships when controlled experiments are not feasible or when a treatment is not successfully delivered t ...


Notes


References

* * * * *.


External links

* by
Mark Thoma Mark Allen Thoma (born December 15, 1956) is a macroeconomist and econometrician and a professor of economics at the Department of Economics of the University of Oregon. Thoma is best known as a regular columnist for ''The Fiscal Times'' throug ...
{{DEFAULTSORT:Consistent estimator Estimator Asymptotic theory (statistics)